Anti-gaming hardening (un-gameable plan) — tracking PR - #91
Merged
Conversation
- benchmark_blind_forgeable(): flags a non-content-whitened active_benchmark.json where a blind score=+-token_id sweep beats chance+3sigma (the deployed 0.785 vs 0.2 break — targets are systematically small-id vs uniform distractors, so a model that learned nothing wins the benchmark leg). make_placeholder_examples is already content-whitened; the DEPLOYED file predates it and must be regenerated. - first commit of the anti-gaming hardening series (un-gameable plan). Next: neutralize benchmark_gain in the crown until the host-reduced benchmark is enforced, then regenerate the deployed file + startup self-test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ENCHMARK_CROWN toggle
…a forgeable file)
…d neutralized-default test
- download_one returns status ('ok'/'decrypt_failed'/'unavailable'), not bool
- decrypt-failed (wrong/rotated seal key) is permanent -> stamp done so it
stops re-downloading + re-failing the same bundle every epoch (was 512
repeated decrypt errors / ~184 retries per dead bundle)
- close the miner's HF PR with the current validator pubkey so they re-seal
- transient (network / mid-upload) stays un-stamped -> retries next epoch
- PR-close is best-effort; a perms/network failure never blocks dedup
- tests: dedup routing + close message carries pubkey + error-swallow + noop
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- SIGALRM wall-clock cap (_HF_FETCH_TIMEOUT_S, default 300s) around list_repo_files + hf_hub_download; huggingface_hub has no total timeout, so a stalled mid-stream fetch froze the whole epoch loop ~35min - timed-out fetch returns 'stalled' (transient) instead of hanging - poll_hub retries a stall, then marks it done after _MAX_STALLS (3) so one unfetchable/slow bundle can't re-stall the loop every epoch - closes a DoS vector: a miner could upload a deliberately-slow bundle - tests: timeout fires + fast passthrough (+ alarm restored) + stall counter Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…LabsAI#91 gaps) - extract _benchmark_selftest: now REFUSES startup (SystemExit) when active_benchmark.json is blind-forgeable AND RALPH_BENCHMARK_CROWN=1 (was warn-only); default crown-off path still just warns - extract _run_idle_audit: same RALPH_AUDIT_LOOP_OFF gate + run_pending_audits call, now a unit-testable helper instead of inline in main() - tests: refuse-on-crown+forgeable, warn-when-crown-off, missing-file noop; audit gate on/off + error-swallow Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Public, incremental hardening of netuid-40 toward EV(cheat)<EV(honest), from the researched plan. Un-gameable is impossible (CC attests load-time not execution; PoL is spoofable; membership-inference cant hit low FPR) — this makes cheating detected + economically deterred, and closes live holes.
Landing here incrementally (watch this space):
Marketing note: this is an attested lineage substrate + economic deterrence, not proof-of-training.